Interactive Experiment: Intro

We challenge you to beat our participants!

Grab a pen and paper! In this experiment demo you get to experience what our research is about in an interactive way. In the following tabs we have included short versions of the different parts in our survey.

First, you get to make a mini Gold-MSI test. The result places you on the musical sophistication scale. The higher your score, the more knowledge and experience you have with music.

The second part consists of a few songs that we used in the survey. It is up to you to determine the valence value of each song to the best of your ability! For both the Gold-MSI test and the song estimation test, the results are on the subsequent tab.

How to grade yourself is explained on the answer tabs. After you have determined your scores, you can see in the graph where you would have approximately been in our sample and if you have surpassed the participants. Good luck!

Mini-MSI

I spend a lot of my free time doing music-related activities.

1 — 2 — 3 — 4 — 5 — 6 — 7

I can compare and discuss differences between two performances or versions of the same piece of music.

1 — 2 — 3 — 4 — 5 — 6 — 7

When I sing, I have no idea whether I’m in tune or not.

1 — 2 — 3 — 4 — 5 — 6 — 7

At the peak of my interest, I practiced ___ hours per day on my primary instrument.

0 — 0.5 — 1 — 1.5 — 2 — 3-4 — 5 or more

Musical Fragments

What do you think is the valence of these four musical fragements? These values range from 1 to 7, with 1 meaning extremely low valence, 7 - extremely high, and 4 - neither high nor low.

SongID: 40

SongID: 80

SongID:155

SongID: 202

Answers and Grading

First, the Mini-MSI results.

Each answer is worth a maximum of 7 points. For question 1, 3 and 4, the first option is worth 1 point, the second 2 points,… For question 2, the scores are reversed, with the first option being worth 7 points, the second 6 points,… Multiply your score by 4,5 to see what your score would’ve roughly been on the full test.

For the music we give the correct valence values. It is up to determine how far you were off or whether you were exactly correct.

On the following tab you can view the raw data of our participants, and you can see how well they did on the songs you listened to by searching for the songs’ IDs (40, 80, 155, 202) in the searchbar

Materials

Song pool: the song pool contains 240 songs, with 20 songs per genre. The genres include: pop, rock, metal, electronic, dance, house, hip-hop, singer/songwriter, soundtrack, R&B, soul/blues and classical music.

Randomized playlist: from the song pool, 25 songs are randomly selected for each participant to listen to and rate.

Gold-MSI: a questionnaire used to measure musical sophistication. Calculated as total number of points on the measure. Only the short version of the questionaire is used, i. e., only the items that map onto the general musical sophistication factor.

Trial round: prior to the main valence estimation task, participants are presented with two songs which they rate the valence of, and are then provided with the true valence of said songs. This is used to help the participants better conceptualize what valence is, and what the main task will look like.

Valence rating: a 7-point Likert scale, ranging from “Extremely low” (1) to “Extremely high” (7), used to estimate a song’s valence.

Familiarity measure: a measured used to indicate whether the participant is familiar with the song that they are rating. This is used in later analysis to see if familiarity affects participant’s judgements of a song’s valence. Measured as 1 (familiar) or 0 (unfamiliar).

R Shiny web app: the web app is used to construct a randomized playlist from the song pool, administer the Gold-MSI, and collect valence and familiarity data.

Song selection

In total, 242 songs were used in this experiment.

Two songs were the same for each participant, namely the trial round songs. These were selected to help participants better conceptualize musical valence. The first song had an extremely high valence value (7), whereas the second song was closer to the middle of the scale and had a slightly low value (3).

The selection process for the other 240 songs began with a breakdown of musical genres - we chose 12 genres based on a survey that was conducted using a sample of 19000 people between the age of 16-64, measuring the 12 most consumed genres of music in 18 countries. (IFPI, 2018). For each genre 20 songs were selected making use of the website ‘rateyourmusic.com’, where albums can be sorted by their average user-submitted rating, by genre. We sampled songs from these highly-rated albums, working under the assumption that well-regarded albums in a given genre are the most representative of said genre. Each sample of songs contained only one song per artist, in a given genre.

During the selection process, we also took note of the valence of the sampled songs. We made sure that - per genre - no valence value was overrepresented or underrepresented, given what is usual for a specific genre; e. g., if a genre is generally characterized as having songs with higher valence, the song pool for said genre would be skewed towards, on average, higher values as well (compared to a song pool of a genre that is characterized as having lower valence songs).

Once the songs were selected, we trimmed each musical item’s length. As a rule of thumb, we chose the 15 seconds from one minute of playback into the song. Some of the selected songs started with an intro, however, this did not count towards the minute.

Procedure

Participants begin the experiment by completing the Gold-MSI to collect measures for musical sophistication. Participants are then directed to the main experiment, where they are first presented with a practice round. In the practice round, participants listen to two songs, and are then asked to rate each song’s valence. After each rating in the practice round, participants are presented with the song’s true valence as determined by Spotify. After this, participants proceed to the main task, which contains the randomized playlist. After listening to a song from the playlist, participants rate the valence using a 7-point Likert scale, and indicate whether they are familiar with the song.

Violin: Valence Estimation Error Density

Jitter: Musical Sophistication & Valence Estimation by Genre

Scatter: True Valence & Estimated Valence

Bar: Total Estimated Valence Error by Genre

Violin Interpretation

The violin graphs depict the density of estimated valence error responses per genre. This plot consists of two parts: one is the box plot, the other is the ‘violin’. From the box plot we can see the minimum and maximum responses, as well as the first quartile, median and third quantile. The thickness of the violin indicates the number of observations for a particular valence deviation.

From this graph we can deduct whether the valences in particular genres were systematically being over- or underestimated by participants. We see that for five genres this is not the case: Electronic, Hip-Hop, Pop, R&B and Singer/Songwriter all have a median of 0. For the other genres however, the medians range from -1 to +2. The most significant being Dance and House with an average of two points estimated above the true valence. For Soundtracks we see that the first quantile and median are equal to +1, but the third quantile is +4.

Jitter Interpretation

In this jitter plot, the valence estimation error (i.e. the difference between the true valence and a participant’s estimation) is plotted against the Gold-MSI score and grouped per music genre. We made use of linear regression to investigate the potential relationship between the dependent and independent variable per genre. It is important to note that, as opposed to the plot on the main page, every dot in this graph represents one attempt of estimating the valence, not one participant. Because this means that we have multiple observations per MSI-value, the command geom_jitter() was used to avoid a straight line of points and make the graph more intuitive to interpret.

As becomes evident from looking at the graph, there seems to be no significant negative relationship between the valence estimation error and musical sophistication for any music genre like we hypothesized. Only the fitted lines for the genres pop, soul/blues and R&B show a very small negative slope. In fact, the genres hip-hop, rock and house even display a small positive relationship. Another interesting finding is that the genres house, dance and soundtracks seem to be systematically overvalued, as indicated by the fitted lines well above 0. This is in accordance with the findings from the violin plot and constitutes an interesting topic for further research.

Scatter Interpretation

As can be seen, some of the genres have similar graphs, but overall, the graphs differ much from each other. A single graph in which all genres are included would not lead to a concise result, therefore we opted for genre-specific graphs. For most genres there is a positive trend, which is most accurately shown by the data for R&B and Singer-Songwriter. Another noteworthy result is that for Dance and House the lower valence values are overestimated on average, for Metal the higher valence values are underestimated on average. For Electronic, Hip-Hop and Rock most average estimates are around the central value, which either indicated a central tendency response pattern, or that for these genres the positivity-grade is not clearly distinguishable.

What also can be seen is that for most genres, songs with extreme-valued valences, so 1 or 7, are not recognized by all participants the song was tested on. These results raise the question whether the valence values determined by Spotify are actually representative for the valence or positivity it evokes in people. The Soundtracks graph raises the same question, because Spotify determined all but one track to have valence value 1 or 2, but the on average judgement by the participants is very spread out.

Bar Interpretation

The bar charts depict the total valence error per genre. The bars are centered around 0, so it is possible to evidently see whether a genre is generally over- or underestimated. Each ‘slice’ in a bar represents an observation of a song in that genre. Moreover, The bars are colored by the estimated valence error, and the thickness of each ‘slice’ correspons to this error, e. g., if a song in the Rock genre was overvalued by 5, then the song’s ‘slice’ is placed in the positive half of the column and has a thickness of 5 units. This means that if an observation was correct, i. e., had an error value of 0, then that slice is not depicted in the graph - this makes it easier to visually inspect whether a genre is generally over- or underestimated.

In accordance with the violin and jitter plots, it seems that songs in the Classical, Dance, House and Soundtracks genres were systematically overestimated. Furthermore, a valence error of 4-6 comprised an evidently larger part of the Classical, Dance, House and Soundtracks genres. When looking at both the scatterplot and bar chart, this suggests that Classical and Soundtracks are underestimated by Spotify, and that Dance and House are overestimated by people (this might be due to the energetic nature of songs in these genres).